Looking for trouble: Using causal semantics to direct focus of attention

Vision should provide an explanation of the scene in terms of a causal semantics|what aaects what, and why. An important part of the causal explanation of static scenes is what supports what, or, counterfactu-ally: Why aren't things moving? We use simple naive physical knowledge as the basis of a vertically integrated vision system that explains arbitrarily complex stacked block structures. The semantics provides a basis for controlling the application of visual attention, and forms a framework for the explanation that is generated. We show how the program sequentially explores scenes of complex blocks structures, identiies functional substructures such as arches and cantilevers, and develops an explanation of why the whole construction stands and the role of each block in its stability. Much work in vision has taken as a central principle the notion that the task of vision is primarily one of explanation (e.g., Witkin & Tanenbaum 83]). However, most of this work has assumed that what needs to be explained is the image data. That is, the task of vision has been assumed to be the construction of a scene model that can explain the image data via rules of image formation, i.e. principles of projective geometry, illumination, reeectance, and so on Barrow & Tanenbaum 78, Horn 86]. The notion that the scene model itself requires explanation, that there are constraints on such explanations quite independent of the rules of image formation, and that these constraints can play an important role in visual perception, has been, if not entirely absent, at least not particularly emphasized. We would like to propose an alternative characterization of what constitutes visual understanding. The cornerstone of our proposal is that visual understanding is fundamentally a matter of developing a causal explanation of the scene; i.e. of determining the causal signiicance of the elements in a scene, and the causal Yarbus 67] BUSTER's attentional trace Figure 1: One of the key determinants of focus of attention is the information one needs to ascertain that a scene makes sense, i.e. that relationships between scene elements are reasonable. Psychologists often study attention in terms of the saccades that subjects make while exploring a scene. For comparison , we show a trace produced by our system while explaining the integrity of a tower of toy blocks. relationships among those elements. Every picture tells a story: Visual understanding consists of guring out what that story is. Causality is central …